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METHODS AND SYSTEMS FOR COMBINING A PLURALITY OF 

RADIOGRAPHIC IMAGES 

CROSS-REFERENCES TO RELATED APPLICATIONS 
[01] The present application claims benefit of U.S. Provisional Patent 

Application S.N. 60/308,997, filed July 30, 2001, entitled "Methods and Systems for 

Combining a Plurality of Radiographic Images," the complete disclosure of which is 

incorporated herein by reference. 

[02] The present invention is also related to U.S. Patent Application S.N. 

09/908,466, filed July 17, 2001, the complete disclosure of which is incorporated herein by 

reference. 

BACKGROUND OF THE INVENTION 
[03] In the medical imaging field, oftentimes the field of view of the 
imaging devices is smaller than the anatomy being examined. Consequently, two or more 
individual images need to be obtained and then properly assembled to form the appropriate 
field of view for analysis. Such assembly of the images is referred to hereinafter as 
"stitching." 

[04] The need for stitching is encountered in many digital radiography, 
MRI, ultrasound, and nuclear medicine evaluations, all techniques that are capable of 
imaging along the axis of possible motion. Unfortunately, stitching of the images is not 
always straightforward. Because it is not always known how much the patient or the imaging 
device has moved or how much the patient shifts or rotates between image shots, accurate 
stitching of the individual images often proves difficult. Thus, flexibility of the stitching the 
images is desirable. 

[05] One particular use in which stitching is often used is in a scoliosis 
evaluation. Scoliosis is defined as a substantial lateral curvature of the vertebral column that 
usually has its onset during periods of rapid growth. Scoliosis curve is determined to be 
present when a structural vertebral column curve of 1 1 ° or more is measured in the coronal 
plane roentgenogram of the erect patient. Radiologic imaging of the spine has traditionally 
been used in the identification, classification, and monitoring of scoliosis. Early detection 
and bracing treatment of juvenile and adolescent idiopathic scoliosis has decreased the need 
for surgery. 



[06] In scoliosis evaluations it is often necessary to stitch the radiographic 
image of the thoracic and upper lumbar spine with the radiographic image of the lumbar and 
lower thoracic spine to provide a large enough field of view to allow the physician to measure 
the angle of scoliosis or the "Cobb angle." Unfortunately, conventional "stitching" methods 
of drawing and measuring directly on the radiographic film have been found to be inaccurate, 
and sometimes introducing errors of ± 5° - 10°, or more. 

[07] Such large alignment errors can affect the perceived alignment of the 
anatomy and dramatically affect the choice of treatment of the patient. For example, when 
the angle of scoliosis is mild (0° - 20°), the recommended treatment is observation and 
careful follow-up. For moderate scoliosis (20° - 40°), bracing is recommended, while severe 
scoliosis (greater than 50°) surgical fusion of the spine is recommended. Thus, the 
physicians' evaluation and the choice of treatment is highly dependent on the evaluation of 

the stitched image. Unfortunately, because the conventional stitching methods can introduce 

deviations of ± 10° or more, the measured angle of scoliosis from the stitched image would 
likely not accurately indicate to the physician how severe a case of scoliosis was present in 
the patient. 

[08] Accordingly, what are needed are methods, software, and systems that 
provide an accurate means for stitching images. It would also be desirable to provide a 
highly versatile set of choices that can increase the ease of stitching. It would further be 
desirable to provide improved quality of the stitched image, especially in the overlap section 
of the stitched images. 

BRIEF SUMMARY OF THE INVENTION 
[09] The present invention provides methods, software, and computer 
systems for stitching radiographic images to form a single, larger field of view radiographic 
image. 

[10] In one aspect, the present invention provides an improved digital 
stitched image. The stitched images of the present invention include a first image and a 
second image. The first image and second image are marked and overlapped so that the 
markers on the first image and the second image are matched together. Because the images 
are overlapped, a portion of the first image and second image are superposed. To improve 
visualization - and to improve the physicians' ability to accurately diagnose the patient - the 
overlapped section of the stitched image can be blended. Proper blending allows for an easier 



understanding of the anatomy and of the features that may have been introduced by motion of 
the patient between shots. 

[11] Each of the digital images of the present invention is composed of a 
plurality of pixels. The pixel intensity for the pixels in the first image and the second image 
5 are calculated. The pixel intensity of the superposed pixels of the first image and second 
image are compared to each other and some function of the pixel intensity of the first image 
and second image can be generated to create the pixels in the blended overlap section. 

[12] For example, in one embodiment, each pixel of the first image in the 
overlap section and each of the corresponding superposed pixels in the second section are 
10 calculated, and the larger of the two measured pixel intensities is displayed. Alternatively, 
the smaller of the two measured pixel intensities is displayed. In yet other embodiments, an 
average of the measured pixel intensities is displayed. 

[13] In yet further embodiments, a smooth transition from the first image to 
the second image can be created. In exemplary embodiments, the transition or overlap 

□15 section includes a first end, a middle point, and a second end. The first end can have 100% of 

Lp 

j; the pixel intensity of the first image and 0% of the pixel intensity of the second image. The 
^ middle point can have 50% of the pixel intensity of the first image and 50% of the pixel 
§ intensity of the second image. The second end can have 0% of the pixel intensity of the first 

L image and 1 00% of the pixel intensity of the second image. Between these points, the 
M20 weights can vary linearly or by some other non-linear weighting relationship. Such an 
Q overlap section should provide an unnoticeable, smooth transition between the first image 
and the second image. 

In another aspect, the present invention provides methods of aligning a 
plurality of images. The methods include the steps of providing a marker on a first image and 
25 a second image and overlapping the first image and the second image to match the marker on 
the first image with the marker on the second image. An absolute difference value is 
computed between the pixel intensities of the overlapping portions of the first and second 
images to validate alignment between the first and second images, 

[14] Advantageously, the methods of the present invention provide direct 
30 visual feedback in real time regarding the registration between the images in the overlap 
section. The absolute difference will display how well the pixels of the overlapped images 
correlate. If there is an exact match, the entire overlap section will be black and the user will 
know that there was a registration. More likely however, there will be some differences in 
the images, due to patient movement or the like, between the images. If the images are not 
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correctly aligned, the user can rotate or translate at least one of the images - observing and 
minimizing the differences in the overlap section - until the images are accurately aligned in 
the area of interest, or over the anatomic feature of interest, even if this is not possible over 
the whole region of overlap. 

[15] After the first and second images are registered, any of the blending 
methods of the present invention can be used to blend the overlap section of the stitched 
image. 

[16] In another aspect, the present invention provides methods of stitching 
and blending at least a first and second image. In exemplary embodiments, the methods of 
the present invention provide a blending that improves the visualization of the stitched image 
in an overlap or transition zone between the first and second images. The methods include 
the step of marking the first and second image with a marker. Typically, the marker will be 
placed over a rigid anatomic marker that is viewable in both the first and second images. A 
portion of the first radiographic image and a portion of a second radiographic image can be 
overlapped so as to match up the markers on the first and second images. To improve 
visualization of the composite image, the overlap section can be blended. 

[17] The radiographic images of the present invention are composed of 
pixels having a pixel intensity that reflects the imaged anatomy. When a first and second 
image are overlapped, the present invention can measure the pixel intensity of the first and 
second images and use the pixel intensity measurements to create and display a blended 
overlap section. 

[18] The present invention provides software and methods which allow the 
user to choose which type of blending method is used to blend the overlap section of the first 
and second image. Allowing the user to select the method of blending provides the user the 
flexibility to select the type of blending that best meets the imaging needs for that particular 
image dataset. The blended overlap section can be created in a variety of ways. Some 
methods of creating the blended overlap section include, but are not limited to maximum 
intensity projection (MIP), minimum intensity projection (MinIP), average, smooth 
transition, and the like. Depending on the desired effect on the image, in exemplary 
embodiments the user will be given a choice as to which blending method is used. For 
example, if the user wants to highlight high density objects, the user can select a maximum 
intensity projection. If the user wants to minimize some undesirable high intensity artifacts 
introduced by the imaging device, the user can select the minimum intensity projection. For 
instance, notice Figure 24, because of edge artifacts in the original images, the Maximum IP 



preserves these and they can be seen as thin wedges in the central portion of the stitched 
image. In contrast as shown in Figure 25, the Minimum IP does not show these wedges, but 
notice that some rib structures (which are not important in this case) are lost along with the 
artifacts. An operator needs both capabilities so as to be able to choose which better fits the 
clinical problem at hand. If the user desires to be aware of relative motion of the subject the 
user can select an average, which then would show blurring where there was motion. If the 
user desires a smooth transition between the first and second image, the user can select a 
blending method which is a weighted average, in which the weighting changing as a function 
of position. 

[19] In another aspect, the present invention provides a method of stitching 
a first image and a second image. The methods include providing a first image and a second 
image. A portion of the first image is overlapped with a portion of the second image. The 
pixel intensities of the first image and second image are calculated validating alignment of 
the first image with the second image by displaying an absolute difference between the pixel 
intensities of the first image and the second image in the overlap section. In such methods, 
the user can visually determine, in real-time, if the first and second images are correctly 
aligned. 

[20] In yet another aspect, the present invention provides methods of 
stitching a first and a second image. In exemplary embodiments, the methods allow the user 
to choose the types of markers that are used to mark the images. Typically, the user will be 
given the choice of how many and/or the type of markers are used to mark and align the first 
and second images. The markers include, but are not limited to, a single point marker, two 
point marker, a line, and a line and a point marker. The present invention moves the images 
so as to match the points as closely as possible, introducing both displacement and rotation to 
achieve this. 

[21] Because anatomic landmarks are variable, having only one type of 
marker available to mark and align the images may not be sufficient to accurately stitch 
images together. A plurality of markers, some of which are suited to different conditions 
better than others, provides the user flexibility to handle the different anatomic landmarks 
that may be visible in the radiographic images. 

[22] In an exemplary embodiment, the present invention marks two points 
on each image. The present invention allows movement of the images so as to match at least 
two of the points. At least one of the images can be rotated with respect to the other so that 
the two points in each image match. Such a method is commonly used for its simplicity. 



[23] In another embodiment, the present invention marks one point on each 
image. The present invention moves the images so as to match the points and keeps the 
orientation of the images fixed. The present invention marks one point on each image when 
it is known that rotation has not occurred, this is a simplification of marking two points on 
5 each image to avoid operator-introduced rotation. 

[24] In another embodiment, the present invention marks one point and a 
line on each image. The present invention matches the points and rotates one image about 
that point so the lines will be parallel. Such a method is useful when one point in the image 
is easily identified, and a long feature (such as a steel brace) is present providing a long 
1 0 region that is easily identified. 

[25] In another embodiment, the present invention marks a line on each 
image. The present invention will match the last point of the first line to the first point of the 
second line and rotate the images to make the lines parallel. This method is useful when a 

P 

P feature such as a brace or a particular bone is seen partially in one image and partially on the 
jJ5 other, with just a minimum of overlap. 

ill 

4- I 26 l In Y et another aspect, the present invention provides methods for 

LU scoliosis analysis. In particular, the present invention provides graphical tools that can 
■ calculate and display the angle of scoliosis (e.g., the Cobb angle), also with real-time 
Lk graphical feedback. Applicants have found that the methods of the present invention 
jip introduce at most only a 1° error into the evaluation of the Cobb angle. Thus, unlike 
Q conventional stitching methods, the physician will be able to accurately determine if the 
patient has a mild, moderate, or a severe case of scoliosis. 

[27] In exemplary embodiments, such stitching can allow for evaluating and 
measuring scoliosis, which involves the computerized stitching of a radiographic image of 
25 the thoracic and upper lumbar spine with a radiographic image of the lumbar and lower 
thoracic spine. While the remaining discussion focuses primarily on the stitching of 
radiographic images for use in scoliosis evaluation, it should be appreciated by those of 
ordinary skill in the art that the present invention can be used to stitch radiographic images 
for a variety of other medical and non-medical purposes. Such purposes include, but are not 
30 limited to MRI, the stitching of coronal or sagittal images obtained in separate sequences, 
such as those acquired to evaluate the spine or the vasculature, in CT for stitching of coronal 
or sagittal images reconstructed from axial projections, in cases where because of scanner 
limitations more than one set has to be acquired to cover the desired length along the long 
axis of the body, and in non-medical applications such as aerial and panoramic photography. 
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[28] These and other aspects of the invention will further evident from the 
attached drawings and description of the present invention. 



BRIEF DESCRIPTION OF THE DRAWINGS 
[29] Figure 1 is a simplified block diagram of a system incorporating the 
5 present invention; 

[30] Figure 2 is a simplified block diagram of an exemplary computer 
system that can run the software of the present invention; 

[31] Figures 3A to 3D illustrate a simplified method of the present 

invention; 

10 I 32 ] Figure 4 a graphical user interface of the present invention displaying a 

single image and a tool palette; 

M , [33] Figure 5 illustrates a graphical user interface showing a first image and 

Q a second image that are to be stitched; 

Q [34] Figure 6A shows a first image having a first marker and a second 

*§5 image having a second marker; 

^ [35] Figure 6B shows a composite image in which the first marker of the 

^ first image is matched up with the second marker of the second image; 

[36] Figure 7A illustrates a first image having a first and second point 
Q marker and a second image having a third and fourth point marker; 
3° [37] Figure 7B illustrates a composite image in which the first and third 

I ** points and second and fourth points have been matched up, respectively; 

[38] Figure 8A illustrates a first image having a first point marker and a 
first line and a second image having a second point marker and a second line; 

[39] Figure 8B illustrates a composite image in which the first and second 
25 point are superposed over each other and the first and second images are rotated until the first 
and second line are in a parallel configuration; 

[40] Figure 9A illustrates a first image having a first line and a second 
image having a second line; 

[41] Figure 9B illustrates a composite image in which a first end of the first 
30 line is aligned with a first end of the second line, and one of the images of Figure 7 A is 
rotated until the lines are parallel; 

[42] Figures 10A and 10B illustrate a method of marking the images using a 
cursor and mouse; 
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[43] Figure 1 1 illustrates a stitched image of the present invention; 

[44] Figure 12 is a flow chart illustrating a simplified method of blending 
an overlap section of superposed first and second images using a maximum intensity pixel; 

[45] Figure 1 3 is a flow chart illustrating another simplified method of 
blending an overlap section of superposed first and second images using a minimum intensity 
pixel; 

[46] Figure 14 is a flow chart illustrating yet another simplified method of 
blending an overlap section of superposed first and second images using an average intensity 
pixel; 

[47] Figures 1 5 A is a simplified view of an overlap section of the stitched 

image; 

[48] Figure 15B is a graph illustrating an exemplary linear smooth 
transition in the overlap section between the first image and second image; 

[49] Figure 1 5C is a graph illustration an exemplary nonlinear smooth 
transition overlap section between the first image and second image; 

[50] Figure 16 illustrates an absolute difference validation method for 
visually indicating mis-registration of the images in the overlap section; 

[51] Figure 17 illustrates software modules of the present invention; 

[52] Figure 1 8 illustrates a zoom icon of the present invention; 

[53] Figure 1 9 illustrates a pan icon of the present invention; 

[54] Figure 20 illustrates a window level icon of the present invention; 

[55] Figure 2 1 illustrates an inverse color icon of the present invention; 

[56] Figure 22 illustrates a Stitch Tool Dialog Menu of the present 

invention; 

[57] Figure 23 illustrates changing the center of rotation of the stitched 

image; 

[58] Figure 24 illustrates a Maximum Intensity Projection Image; and 
[59] Figure 25 illustrates a Minimum Intensity Projection Image. 

DETAILED DESCRIPTION OF THE INVENTION 
[60] The present invention provides improved methods, systems, software 
and graphical user interfaces for allowing a user to stitch and/or blend a plurality of DICOM 
digital radiographic images together. 
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[61] Figure 1 is a simplified block diagram of a system 10 which may 
incorporate the present invention. As shown, system 10 comprises an imaging device 12, 
such as an x-ray, MRI, CT, ultrasound, nuclear imaging device, or the like that is coupled to a 
communication network 14 (such as an intranet, LAN, WAN, or the internet) via 
communication link(s) 16. System 10 depicted in Figure 1 includes a computer system 22 
that communicates with the imaging device that can run software for manipulating images 
obtained from imaging device 16. It should be appreciated however, that system 10 depicted 
in Figure 1 is merely illustrative of an embodiment incorporating the present invention and 
does not limit the scope of the present invention. For example, instead of delivering the 
image data to computer system 22 via a communication network, the images can be delivered 
to the computer software via a computer readable medium, such as a floppy disk, CD-ROM, 
or the like. Alternatively, images obtained from the imaging device can be stored on a 
separate server or database 18 that is coupled to computer system 22 via communication 
network 14. 

[62] Communication network 14 provides a mechanism allowing the 
various components of computer network 14 to communicate and exchange information with 
each other. Communication network itself may be comprised of many interconnected 
computer systems and communication links. Communication links 16 may be hardwired 
links, optical links, wireless links, or other conventional communication links. 

[63] Figure 2 is a simplified block diagram of an exemplary computer 
system 22 that can run the software of the present invention. Computer system 22 typically 
includes at least one processor 28 which communicates with a number of peripheral devices 
via a bus subsystem 26. These peripheral devices may include a storage subsystem 36, 
comprising a memory subsystem 38 and a file storage subsystem 44, user interface input 
devices 34, user interface output devices 32, and a network interface subsystem 30. Network 
interface subsystem 30 provides an interface to outside networks, including an interface to 
communication network 20, and is coupled via communication network 46 to corresponding 
interface devices in other computer systems. 

[64] User interface input devices 34 may include a keyboard, pointing 
devices such as a mouse, trackball, touch pad, or graphics tablet, a scanner, foot pedals, a 
joystick, a touchscreen incorporated into the output device 32, audio input devices such as 
voice recognition systems, microphones, and other types of input devices. In general, use of 
the term "input device" is intended to include a variety of conventional and proprietary 
devices and ways to input information into computer system 24 or onto computer network 46. 



[65] User interface output devices 32 may include a display subsystem, a 
printer, a fax machine, or non- visual displays such as audio output devices. The display 
subsystem may be a cathode ray tube (CRT), a flat-panel device such as a liquid crystal 
display (LCD), a projection device, or the like. The display subsystem may also provide non- 
visual display such as via audio output devices. In general, use of the term "output device" is 
intended to include a variety of devices and ways to output information from computer 
system 24 to an operator or to another machine or computer system. 

[66] Storage subsystem 36 stores the basic programming and data 
constructs that provide the functionality of the various embodiments of the present invention. 
For example, database and modules implementing the functionality of the present invention 
may be stored in storage subsystem 36. These software modules are generally executed by 
processor 28. In a distributed environment, the software modules may be stored on a 
plurality of computer systems and executed by processors of the plurality of computer 
systems. Storage subsystem 36 typically comprises memory subsystem 38 and file storage 
subsystem 44. 

[67] Memory subsystem 38 typically includes a number of memories 
including a main random access memory (RAM) 42 for storage of instructions and data 
during program execution and a read only memory (ROM) 40 in which fixed instructions are 
stored. File storage subsystem 44 provides persistent (non-volatile) storage for program and 
data files, and may include a hard disk drive, a floppy disk drive along with associated 
removable media, a Compact Digital Read Only Memory (CD-ROM) drive, an optical drive, 
or removable media cartridges. One or more of the drives may be located at remote locations 
on other connected computers at other sites coupled to communication network 20. The 
databases and modules implementing the functionality of the present invention may also be 
stored by file storage subsystem 44. 

[68] Bus subsystem 26 provides a mechanism for letting the various 
components and subsystems of computer system 22 communicate with each other as 
intended. The various subsystems and components of computer system 22 need not be at the 
same physical location but may be distributed at various locations within distributed network 
1 0. Although bus subsystem 26 is shown schematically as a single bus, alternate 
embodiments of the bus subsystem may utilize multiple busses. 

[69] Computer system 22 itself can be of varying types including a personal 
computer, a portable computer, a workstation, a computer terminal, a network computer, a 
module in the imaging unit, a mainframe, or any other data processing system. Due to the 
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ever-changing nature of computers and networks, the description of computer system 24 
depicted in Figure 2 is intended only as a specific example for purposes of illustrating the 
preferred embodiment of the present invention. Many other configurations of computer 
system 24 are possible having more or less components than the computer system depicted ir 
Figure 2. 

[70] Figures 3A to 3D illustrate a simplified stitching method that can be 
performed on the systems 10 of the present invention. As shown in Figures 3 A and 3B, at 
least a first image 50 and second image 52 can be stitched together to form a single, 
composite image 54 (Figure 3D) that provides a larger field of view for visualization and 
analysis by an examining physician. In order to accurately perform an analysis on target 
areas of the body that cannot be properly viewed on a single radiographic image, the 
physician must stitch the two images 50, 52 together. As shown in Figure 3A and 3B, the 
target images can be marked with a marker 56, 56' at substantially the same anatomic 
landmark in the patient's body. Typically, the user will place marker 56 over a rigid 
landmark, such as any metal screws, stents, brace, vertebral bodies, joints, or the like. 

[71] Oftentimes, because the subject or imaging device will have moved or 
rotated during imaging, the first and second images may be taken from different angles and it 
may be difficult to accurately match the two images. Thus, as will be described in detail 
below, certain marking techniques may be more beneficial than other methods of marking in 
stitching the two or more images together. 

[72] As shown in Figure 3C, after the images are marked, markers 56, 56' 
on the first image 50 and second image 52 can be matched together automatically with the 
software. In some embodiments, the images can be translated and superposed without any 
rotation. In other embodiments, however, the images can be translated and rotated so that the 
markers can be accurately aligned. As shown in Figure 3C, when the markers on the first and 
second images are superposed over each other, a portion of the first image and a portion of 
the second image will also be superposed over each other. Such sections are hereinafter 
referred to as an "overlap section 58." In exemplary embodiments, the present invention 
provides methods of blending the overlap section 58 so as to provide improved visualization 
of the composite stitched image. Once the overlap section 58 is blended the final stitched 
image 54 can be analyzed by the examining physician. 

[73] Referring now to Figure 4, the present invention further provides a 
graphical user interface 60 that can be displayed on output device 32 of the computer system 
22 of the present invention (Figure 2). The graphical user interface 60 of the present 
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invention has an image window 62 and a tool palette 64 for allowing user input to manipulate 
the images. As illustrated by the pull-down menu 66, the stitching functionality (shown 
herein as "AccuStitch") provided by the present invention can be a software module of an 
imaging software program, or alternatively the stitching functionality can be a stand alone 
software program. Another exemplary graphical user interface that can incorporate the 
present invention is described in U.S. Patent Application S.N. 09/908,466, filed July 17, 
2001, the complete disclosure of which is incorporated herein by reference. 

[74J When the user desires to stitch a plurality of images together, the user 
can download the desired images into the software, either via communication network 14 or 
from memory 36, and display the images in the image window 62 of graphical user interface 
60. As shown in Figure 5, first image 68 and second image 70 are typically displayed 
adjacent to one another in image window 62. Typically, first image 68 and second image 70 
are taken from the same imaging unit and have the same dimensionality and gray scale. If 
desired, the user can manipulate the zoom factor of the images, translate the images, switch 
the images, rotate the images, or the like, using tool palette 64. After images 68, 70 are 
displayed in image window 62, the user can place at least one marker/fiducial on each of the 
images over the same anatomic landmark(s), blend the overlap section of the image, and the 
like. 

[75] In exemplary embodiments, the present invention can allow the user to 
choose what type of marker or fiducial can be placed on the images to mark and align the 
images. Applicants have found that providing a plurality of marking methods gives the user 
the ability to accurately align the images, no matter what type of anatomic conditions are 
present in the image dataset. In some situations, it may be desirable to use a single point to 
align the images. In other instances, however, it may be required to use a plurality of markers 
(e.g., two points, a point and a line, etc.) on each image to correct any rotation or movement 
of the patient during the imaging. Some exemplary marking methods are discussed herein 
below. 

[76] As shown in Figure 6A, if it is known that the patient has been held 
rigid and the imaging device is held rigid such that there is no rotation of the target tissue 
when the images are obtained, a single point will be the easiest and fastest method of aligning 
the first and second images. As shown in Figure 6A, a first marker 72 can be placed 
over/adjacent a landmark 74 in first image 50, and a second marker 76 can be placed 
over/adjacent landmark 74' in second image 52. After the images 50, 52 have been marked, 
the markers can be superposed over each other so as to align images 50, 52 (Figure 6B). 
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When only one marker is used to mark the images, typically the first and second images will 
not be rotated so as to prevent the introduction of mis-registration of the images. It should be 
appreciated however, that in alternative embodiments, it may be possible to rotate at least one 
of the images to properly align the first image 50 with the second image. 

[77] Due to patient breathing and/or shifting during the imaging, oftentimes 
there will be some misalignment between the first image and the subsequent images taken 
due to movement of the patient during imaging. Thus, a single point marker may not be 
sufficient to accurately align and stitch the images together. Figure 7A shows a first image 
50 5 and a second image 52' in which the user places a first and second marker 76, 78 on the 
first image 50' and a third and fourth marker 80, 82 on the second image 52\ To stitch the 
first and second image, the user can manually align or the computer software can be 
programmed to automatically align the first marker 76 with the third marker 80 and the 
second marker 78 with the fourth marker 82 (Figure 7B). Similar to the method of Figure 
6 A, the markers are typically positioned an adjacent anatomic landmark that is visible in both 
the first image 50' and second image 52'. 

[78] In placing two markers on each image the operator may inadvertently 
introduce a small placement error, so that the distance between the markers in the first image 
is not equal to the distance between the markers in the second image. In that case, the 
software splits the difference, and further allows for small manual adjustments for further 
refining the position of the markers. 

[79] Figure 8 A shows a first marker 84 and a first line LI on a first image 
50" and a second marker 86 and a second line L2 on a second image 52". In such a marking 
method, the user will place first marker 84 and second marker 86 substantially over the same 
anatomic landmark in the first and second image. First line LI and second line L2 can be 
positioned in the first and second image. To align and stitch the first and second images, as 
shown in Figure 8B, the user or computer software can superpose the first marker 84 over the 
second marker 86, or vice-versa. Thereafter, at least one of the images 50", 52" can be 
rotated until lines LI and L2 are parallel. The image will typically be rotated about points 84, 
86 until the lines LI, L2 are parallel. 

[80] Figure 9A illustrates a method of aligning a first image 50"' with a 
second image 52"' through use of lines. A first line 88 having a first point 90 and a second 
point 92 is drawn on the first image. A corresponding second line 94 having a third point 96 
and a fourth point 98 can be drawn in a contiguous anatomic position in the second image. 
The software aligns point one 90 and point three 96 (or points two and four if desired) and 
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rotate the first line 88 or second line 94 about the first point/second point so that the lines are 
superposed over each other. The stitched image is shown in Figure 9B. 

[81J The point markers and lines can be placed on the images using a 
variety of conventional and proprietary methods. For example, in one exemplary method, a 
user can click on a button of a mouse while placing a cursor 101 over a target portion of the 
image (Figure 10A). To draw a line 103, the user need only left click on a mouse button at a 
point of the image to start the line and drag the cursor 101 to an ending point 105 of the line 
and release the mouse button (Figure 10B). It should be appreciated however, that other 
input devices, such as the keyboard, joystick, or the like can be used to draw the points and 
lines. 

[82] Providing a plurality of marker types allows the user to align 
radiographic images that may not be accurately aligned if only one type of marker is 
provided. Such flexibility of marking the images improves the visualization of the target 
body area, and consequently provides the examining physician an opportunity to accurately 
examine the images. 

[83] The present invention marks one point on each image when it is known 
that rotation has not occurred, this is a simplification of marking two points on each image to 
avoid operator-introduced rotation. 

[84] In another embodiment, the present invention marks two points on 
each image. The present invention moves the images so as to match the points and rotates 
one image with respect to the other so that the two points in each image are match. Such a 
method is commonly used for its simplicity. 

[85] In another embodiment, the present invention marks one point and a 
line on each image. The present invention matches the points and rotates the image about 
that point so the lines will be parallel. Such a method is useful when one point in the image 
is easily identified, and a long feature (such as a steel brace or leg bone) is present providing 
a long region that is easily identified. 

[86] In another embodiment, the present invention marks a line on each 
image. The present invention will match the last point of the first line to the first point of the 
second line and rotate the images to make the lines parallel. This method is useful when a 
feature such as a brace or a particular bone is seen partially in one image and partially on the 
other, with just a minimum of overlap. 

[87] After the first image 50 and second image 52 have be superposed and 
aligned with each other, in some exemplary methods and software of the present invention, 
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the user can blend the overlap section 18 of the images together to improve visualization of 
the overlap section 18 (Figure 1 1). The software of the present invention will typically 
support a plurality of methods to blend the overlap section of the images. Each of the 
methods of blending can be used to provide different visual outputs, depending on the desired 
characteristics of the overlap section. The blended overlap section can be created in a variety 
of ways. Some methods of creating the blended overlap section include, but are not limited 
to maximum intensity projection (MIP), minimum intensity projection (MinIP), average, 
smooth transition, and the like. Depending on the desired effect on the image, in exemplary 
embodiments the user will be given a choice as to which blending method is used. For 
example, if the user wants to highlight high-density objects, the user can select a maximum 
intensity projection. If the user wants to minimize some undesirable high intensity artifacts 
introduced by the imaging device, the user can select the minimum intensity projection. If 
the user desires to be aware of relative motion of the subject the user can select an average, 
which then would show blurring where there was motion. If the user desires a smooth 
transition between the first and second image, the user can select a blending method which is 
a weighted average, the weighting changing as a function of position. 

[88] In another aspect, the present invention provides a method of blending 
a first image and a second image. The pixel intensities of the first image and second image 
are calculated and alignment of the first image with the second image is validated by 
displaying an absolute difference between the pixel intensities of the first image and the 
second image in the overlap section. In such methods, the user can visually determine, in 
real-time, if the first and second images are correctly aligned. 

[89] Advantageously, the methods of the present invention provide direct 
visual feedback in real time regarding the registration between the images in the overlap 
section. The absolute difference will show how well the pixels of the overlapped images 
correlate. If there is an exact match, the entire overlap section will be black and the user will 
know that there was a registration. More likely however, there will be some differences in 
the images, due to patient movement or the like, between the images. If the images are not 
correctly aligned, the user can rotate or translate at least one of the images observing and 
minimizing the differences in the overlap section until the images are accurately aligned in 
the area of interest, or over the anatomic feature of interest, even if this is not possible over 
the whole region of overlap. 

[90] Figure 12 shows a simplified flow chart of one exemplary blending 
method that displays a maximum intensity pixel in the overlap section. In such methods, a 
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pixel intensity of all of the pixels in overlap section of the first and second images is 
calculated on a pixel-by-pixel basis (Steps 100, 102). The pixel that is actually displayed in 
the overlap section will be the larger of the measured pixel intensity of the overlapping 
corresponding pixels in the first image and the second image (Step 104). Such a method 
provides a maximum sharpness and reduces any blurring due to movement of the patient 
during imaging and highlights high-density objects (Figure 24). 

[91] Figure 13 shows a simplified flow chart of another exemplary blending 
method that displays a minimum intensity pixel in the overlap section. In such methods, a 
pixel intensity of all of the pixels in overlap section of the first and second images is 
calculated on a pixel-by-pixel basis (Steps 106, 108). The pixel that is actually displayed in 
the overlap section will be the smaller of the measured pixel intensity of the overlapping 
corresponding pixels in the first image and the second image (Step 1 10). Such a method 
provides a minimization of some undesirable high intensity artifacts introduced by the 
imaging device (Figure 25). 

[92] Figure 14 shows a simplified flow chart of yet another exemplary 
blending method that displays a pixel having an average intensity value of the pixel 
intensities from the corresponding pixels in the first and second images. In such methods, a 
pixel intensity of all of the pixels in overlap section of the first and second images is 
calculated on a pixel-by-pixel basis (Steps 1 12, 1 14). The pixel that is actually displayed in 
the overlap section will be the average of the measured pixel intensity of the corresponding 
overlapping pixels in the first image and the second image (Step 1 16). Such a method 
provides a means to show blurriness where there was relative motion of the subject. 

[93] The present invention also provides a method of blending the overlap 
section in which the overlap section has a smooth transition between the first image and 
second image. As shown schematically in Figure 15 A, the overlap section 58 has a first 
border or end 120 that is closer to an upper portion of the first image 50 and a second border 
or end 122 that is adjacent the lower portion of the second image 52. First border 120 will 
display a pixel intensity that is 100% from the first image and 0% from the second image. 
Similarly, second border 122 will have a pixel intensity that is 100% of the second image and 
0% of the first image. Typically, the midway point 124 between the first border 120 and the 
second border 122 will be an average intensity of the first image and second image (e.g., 50% 
of the first image and 50% of the second image). For simplicity, Figure 15A shows a first 
image and a second image that are not rotated. It should be appreciated however, that the 
blending methods can be used equally well with images that have been rotated 
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[94] In some embodiments, as shown in Figure 15B, the pixel intensity of 
the points between the midway point 124 and the first and second borders can have a linear 
relationship between the pixel intensity of the corresponding pixels of the first image and 
second image. In other embodiments, as shown in Figure 1 5C the pixel intensity of the 
5 points between the midway point 124 and the first and second borders can have a non-linear 
relationship between the pixel intensities of the first image and second image. 

[95] As shown in Figure 16, the present invention also can provide an 
"absolute difference" method to validate the registration of the image and to provides direct 
visual feedback in real-time so as to enhance the visibility of any mis-registration of the 
10 images. In the absolute difference method the software will subtract the pixel intensity of the 
first image from the pixel intensity of the second image, or vice versa. If the first image and 
second image in the overlap section are perfectly aligned, the user will see only black in the 
overlap section since the absolute difference will be zero. Generally, however, the user will 
5? see some black and non-black pixel intensities in areas that are not perfectly aligned due to 

had 

C15 rotation or translation of the patient during imaging. Thus, the absolute difference method 

y'l 

V will be able to provide a real-time visual indication of when the images are substantially 

C i 

, aligned. If there is a mis-registration, each image may be rotated or translated until an 
s acceptable match is obtained. 

Li, [96] After the first and second images have been aligned and blended using 

§§) any combination of the above described methods, the resulting composite image can be saved 

01 

O as a separate DICOM image file that can be transferred for analysis by the examining 
physician. 

[97] Figure 1 7 depicts the software modules according to an embodiment of 
the present invention. For example, according to an embodiment of the present invention, 
25 software modules implementing the functionality of the present invention may be stored in 
storage subsystem 36 (Figure 2). These software modules may be executed by processor(s) 
50 of computer system 22. In a distributed environment, the software modules may be stored 
on a plurality of computer systems and executed by processors of the plurality of computer 
systems. 

30 [98] An exemplary data flow through the software of the present invention 

will now be described. As shown in Figure 17, the software of the present invention typically 
includes a graphical user interface module 130, a stitching module 132, and an image analysis 
module 134. Image information is obtained with imaging device 12 and stored in memory 36 
or other computer readable medium. The software modules can obtain the image data from 
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the computer readable medium and display the selected images on graphic user interface 60. 
After the first image is displayed on the graphical user interface 60, the user can use the 
graphical user interface module 130 to zoom in or out of the image by multiplying the slice 
dimension by a chosen zoom factor and resample the image from the original dataset. 
5 Moreover, if the user desires to pan the displayed image, using known methods, the user can 
use the graphical interface module 130 to cause a 2D transformation of the image by moving 
the center of the displayed image to a desired point on the graphical user interface. 

[99] After a second image is obtained from the computer storage, the first 
and second images can be marked with a fiducial. The stitching module 132 will typically 

1 0 allow the user to choose which types of marker will be used to mark the images. As 
described above, in certain imaging situations it may be preferable to use one stitching 
method (e.g., one point, two points, lines, or the like) over the other markers. The stitching 

^ module 132 allows the user to select and place a marker on each of the images. After the 

g markers have been placed on each of the images, at least one of the images can be translated 

:J5 and possibly rotated until the markers are substantially aligned. 

jj [100] Image translation can be performed to the image (x, y) by adding 

translation amounts to the coordinates of the points. For the new position of image P f (x, y) , 



y, the following formula can be used to move each point P(x, y) by d x units parallel to the 
q x axis and by d y units parallel to the y axis. 



C x'=x + d x9 y'=y+d y 

[101] Image rotation about a chosen origin, through an angleO can be 
performed to the image (x, y) by the following formula. 

25 

x'= x-cosO — -sin 9 , y=x-sin9 —y -cosG 



[102] After the images have been moved and substantially aligned, stitching 
30 module 132 can be configured to blend the overlap section of the first and second images to 
improve visualization of the stitched image. The stitching module 132 can include a variety 
of blending algorithms so as to allow the user flexibility in choosing how to align and blend 
the first and second images together. 

[103] In a specific embodiment, once the images have been substantially 
35 aligned, five blending formulas can be selectively used to calculate the overlap section. For 
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every point P\x 9 y) inside the overlapped area, one of the following five blending formulas 

can be used to produce the new image from two source images, P x (x,y) and P 2 (x,y) . It 
should be appreciated however, that these formulas are merely examples, and other formulas 
can be used to blend the overlap section. 



• Average - averaged value between two images. 

Pfay^P^y) 



P'(x,y)= 



• Absolute Difference - absolute difference value between two images. 

Pix^PfayyPt&y] 

• Maximum Intensity Projection - on a pixel-by-pixel basis, selects the densest values 
from the two images. 



K ' y) \P 2 (x,y), if P 2 {x,y)>P,{x,y) 



• MinIP - on a pixel-by-pixel basis, selects the least dense values from the two images. 

Pi 



{P2M if P 1 (x,y)>P 2 (x,y) 



Blend - smooth transition between two source images 

M = size of overlapped area parallel to the x axis 0 < / < m 
N = size of overlapped area parallel to the y axis o<j<n 

In the image space where: 

P\xb •> Pixb = Beginning of the source image 1 and 2 parallel to the x axis 
P\xe > Pixe = Ending of the source image 1 and 2 parallel to the x axis 
P\yb > Piyb = Beginning of the source image 1 and 2 parallel to the y axis 
Piye > Piye ~ Ending of the source image 1 and 2 parallel to the y axis 



d x ~ min 



Pixb 
J-Plyb 
Pixe-* 
{Plye~J 



d 2 = min< 



l~P2xb 
J-Plyb 
Pixe"* 
Piye"! 



d x + d 2 



w 2 =-- dl - 



d x + d 2 

p<(i,j)=WiPXhJ)+w 2 p 2 (i,j) 
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[104] After the images have been stitched and blended, the image can be 
stored and the examining physician can examine the image. As noted above, in scoliosis 
evaluation, the examining physician needs to measure the angle of the patient's spine (i.e. 
Cobb's angle). In such uses, the image analysis module 134 can include an algorithm to 
measure the angle of the patient's spine. In a particular embodiment, the user can draw a line 
in the disk space between two thoracic vertebrae parallel to the inferior surface of the upper 
vertebrae and a second line in the disk space between two lumbar vertebrae, parallel to the 
inferior surface of the upper lumbar vertebrae. The program can then automatically draw a 
line perpendicular to each of the two user drawn lines and the image analysis module 134 can 
calculate the angle at the intersection. The measured angle can then be recorded and 
displayed and used as a reference for treatment recommendation. One specific formula for 
calculating the angle between two given lines Z, andZ 2 : 



9 =tan 







J-tan" 1 


( L 

^2y\ 


^2y2 




~ L\x2 j 




" ^2x2 J 



Where: 

L \x\ > L ix\ = St art point of line 1 and 2 parallel to the x axis 
L \x2 > L ix2 = End point of line 1 and 2 parallel to the x axis 
L \ y \ > L 2y\ = Start Point of line 1 and 2 parallel to the y axis 
L \y2 > L 2y2 = End point of line 1 and 2 parallel to the y axis 

[105] Applicants have found that such an image analysis module has 
dramatically reduced the errors introduced into the measurement of the angle of the spine 
(i.e., Cobb angle measurement) such that the introduction of errors into the angle 
measurement was below 1°. It should be appreciated, that the image analysis module 134 can 
contain a plurality of algorithms to measure anatomic conditions in the stitched image. For 
example, in addition to or instead of the line angle measurement algorithm, the image 
analysis module can include means to measure lengths, distances between anatomic features, 
and the like. 

[106] As illustrated in Figures 1 8-23, the present invention provides 
graphical user interfaces for manipulating radiographic images. While the following 
discussion describes one exemplary graphical user interface and methods, it should be 
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appreciated that the present invention can take many different forms that are not described 
herein, and the present invention is not limited to such an example. 

[107] Figure 4 illustrates an exemplary tool palette of the present invention. 
The tool palette can include icons to (1) manipulate the image such as zoom, pan, change the 
window level, inverse color, and (2) tools to stitch the images such as stitch tools and swap 
images, adjust image. It should be appreciated that other combinations of other conventional 
icons can be incorporated into the tool palette without departing from the scope of the present 
invention. All functions available on the tool palette 64 can also be available in a pull down 
menu of the main menu bar. This redundancy allows the user to employ fewer mouse clicks 
to reach frequently used commands. 

[108] As shown in Figure 1 8, selecting the zoom icon 1 50 will magnify the 
image in real time. The zoom factor can be changed by clicking on a left mouse button or by 
depressing selected buttons on a keyboard (typically the up arrow and down arrow). The 
current zoom factor 151 will typically be displayed on the image window to inform the user 
of the current zoom factor. 

[109] As shown in Figure 1 9, the user can select the pan icon 1 52 and pan 
up, down, left, and right within the image so as to display a desired area. The user can use 
the mouse and/or click on the scrollbars 153, 153' along the right and bottom of the image 
window to pan the images. Additionally, the user can select the icon and use the arrow keys 
on the keyboard to pan through the selected image. 

[110] Selection of the window level icon 1 54, as shown in Figure 20 can be 
used to change the contrast and brightness of the image. The window level can be displayed 
on the image window. Adjustment of the window level can be done with the mouse or finely 
adjusted with the keyboard keys. 

[Ill] As shown in Figure 21, selecting the inverse color icon 156 inverses 
the color table from black to white, and vice versa. This allows for preferences and 
conventions in reading images. It is generally recognized that given a certain background 
intensity, bright features are easier to observe than dark ones, and reversing the scale can aid 
the operator who may be choosing either dark or bright anatomic landmarks. 

[112] Selecting the Stitch tools icon 158 (Figure 21) will bring up a Stitch 
Tool Dialog 160. As shown in Figure 22, the Stitch Tool Dialog 160, the user can select the 
type of markers so use to mark the image (e.g., one fiducial, two fiducial, or lines). Selection 
of the type of marker to use will depend primarily on the anatomic landmarks available in the 
images. Selecting one of the icons will allow the user to mark the images, as described above 
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in reference to Figures 6 A- 10. After the fiducials have been placed in the appropriate places 
on the image, the user can actuate the Stitch icon 158 to begin the stitching process. 

[113] After the image has been stitched, the user can adjust the position and 
rotation of the stitched image by activating the Adjust Image icon 162. In exemplary 
embodiments, the image can be moved one pixel at a time using the keyboard - typically the 
left, right, up and down keys. To change the rotation, the user can depress the Page Up and 
Page down keys. If the user depresses the "Shift" key on the keyboard in combination with 
the other keys, the movement will be increase by a factor of ten. Thus, if the Shift key is held 
down while depressing the left key, the image will move ten pixels. Similarly, if the Shift 
key is held down in combination with the Page Down key, the stitched image will rotate ten 
degrees. 

[114] As shown in Figure 23, to move the center of rotation of the stitched 
image, a marker 164 in the picture can be moved by clicking and dragging the marker to a 
new center of rotation. Typically, the marker can be dragged to another position by holding 
the mouse button down and releasing the button when the cross-mark is at the desired center 
of rotation. 

[115] While the above disclosure as described herein is focused on stitching 
digitized radiographic images, it should be appreciated that the present invention is equally 
applicable to CT, MRI, nuclear imaging, ultrasound, aerial and panoramic photography, and 
other image datasets. Additionally, while the above invention is directed mainly to stitching 
of medical images, the present invention can be used for editing digital photographs, and the 
like. 
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